///////////////////////////////////////////////////////////////////////////////
/**
 Lexer Grammar file
 -------------------
 
 Antlr breaks downs the works into different components. Lexer component 
 processes an input char stream into tokens. These tokens are processed by
 Parser component of Antlr. Tokens objects know their 
 start/stop stream index, line number, index within the line, and more. Lexer
 outputs/throws error if an undefined token is found in the input stream.
 
 In this example, we have to define the tokens for a comma separated list of
 integers (E.g. 4,56,98) that can be spread across multiple lines. User can
 also put Java style comments in the middle of this list. For more examples,
 look at TestCSV.java  

 The grammar file consists of a list of instructions. An instruction ends with
 ';'. Comments can be specified using the same syntax as Java.

 Besides defining the token specifications, in this example, we will
 also define grammar actions to customize the generated Lexer Java code.
 */

///////////////////////////////////////////////////////////////////////////////
/*
 * The first instruction should specify the grammar type (lexer) and the
 * name of the grammar (CSVLexer). This name should be same as the name of
 * the grammar file without the extension.
 *
 * AntlrWorks or Ant tasks can be used to create the CSVLexer.java file from
 * this grammar file.
 */
lexer grammar CSVLexer;

///////////////////////////////////////////////////////////////////////////////
/*
 * Instructions starting with '@' are called grammar actions. Grammar actions
 * are used to add code to the generated Java code. 
 */

/*
 * '@header' grammar action is used to specify the package and imported classes.
 *
 * In instruction below, we have specified the package to match our tutorial
 * layout.
 */
@header{package org.concepts.java.antlr.csv;}

///////////////////////////////////////////////////////////////////////////////
/*
 * '@members' grammar action is used to specify Lexer class member variables and
 * additional methods. Lexer behavior can also be changed by overriding methods
 * that are inherited from the superclass (Lexer).
 *
 * In the instructions below we add the capability in the lexer to count the
 * number of integers.
 */
@members{

  private int integerCount = 0;

  /**
  * Returns the number of integers in the list 
  */
  public int getIntegerCount() {
    return integerCount;
  }
  
} // @members

///////////////////////////////////////////////////////////////////////////////
/*
 * Token specification. The tokens are defined using EBNF
 * http://en.wikipedia.org/wiki/Extended_Backus_Naur_Form .
 * 
 * Six tokens are defined. By convention, all tokens are defined in upper-case.
 *
 * INT : Identify integer.
 * COMMA : Comma.
 * WHITESPACE : Any space or tab.
 * NEWLINE : Line break.     
 * SINGLE_COMMENT: Comment in a single line that starts with //
 * MULTI_COMMENT: Comments that can be split across multiple lines.
 *    
 */

/*
 * Comma token.
 */
COMMA: ',' ;

/*
 * Integer token. All consecurity digits are put into a single integer token.
 * Java code between {..} is run once a given token is processed
 */
INT : '0'..'9'+ {integerCount++;};

/*
 * Whitespace is one or more space (' ') and/or tab ('\t') character. Tokens are
 * passed to the parser components on channels. In this case, we have decided
 * to pass all whitespace on a hidden channel that is not automatically
 * inspected by the parser.
 */
WHITESPACE: (' ' | '\t')+ {$channel = HIDDEN;};

/*
 * Newline is '\n' on most operating systems. In some operating system it's
 * '\r' following by '\n'. 
 */
NEWLINE: ('\r'? '\n')+;

/*
 * Single comment is a line that starts with '//' and ends with a new line.
 * It can consist of zero or more characters, execpt any new line character.
 * By using 'skip();' java code we have instructed the lexer to not process 
 * this token. Thus this token won't be available to the parser. 
 */
SINGLE_COMMENT : '//' ~('\r' | '\n')* NEWLINE {skip();};


/*
 * Multi line comment start with '\/*' and ends with '*\/'. It can consist of
 * any characted in the middle. By default matching of a token is greedy, i.e.
 * the lexer will go till the last '*\/' in the characted stream. We want the
 * lexer to stop at the first '*\/', thus we specify greedy to false in the
 * token option.
 */
MULTI_COMMENT options { greedy = false; } : '/*' .* '*/' NEWLINE? {skip();};

///////////////////////////////////////////////////////////////////////////////
